Validating Bayesian truth serum in large-scale online human experiments
نویسندگان
چکیده
Bayesian truth serum (BTS) is an exciting new method for improving honesty and information quality in multiple-choice survey, but, despite the method's mathematical reliance on large sample sizes, existing literature about BTS only focuses on small experiments. Combined with the prevalence of online survey platforms, such as Amazon's Mechanical Turk, which facilitate surveys with hundreds or thousands of participants, BTS must be effective in large-scale experiments for BTS to become a readily accepted tool in real-world applications. We demonstrate that BTS quantifiably improves honesty in large-scale online surveys where the "honest" distribution of answers is known in expectation on aggregate. Furthermore, we explore a marketing application where "honest" answers cannot be known, but find that BTS treatment impacts the resulting distributions of answers.
منابع مشابه
Constructing a hypothesis space from the Web for large-scale Bayesian word learning
The Bayesian generalization framework has been successful in explaining how people generalize a property from a few observed stimuli to novel stimuli, across several different domains. To create a successful Bayesian generalization model, modelers typically specify a hypothesis space and prior probability distribution for each specific domain. However, this raises two problems: the models do no...
متن کاملLarge-scale Structured Learning
In this thesis we study large-scale structured learning in the context of supervised, unsupervised and semi-supervised learning. In the big data era, it is increasingly important to automatically infer structure from the data or leverage human provided structures in various learning processes. In the first part of this thesis, we focus on how to harness external supervision about the structural...
متن کاملVisual Concept Learning: Combining Machine Vision and Bayesian Generalization on Concept Hierarchies
Learning a visual concept from a small number of positive examples is a significant challenge for machine learning algorithms. Current methods typically fail to find the appropriate level of generalization in a concept hierarchy for a given set of visual examples. Recent work in cognitive science on Bayesian models of generalization addresses this challenge, but prior results assumed that objec...
متن کاملIncentives to Counter Bias in Human Computation
In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wisdom of the crowd. However, human computation is susceptible to systematic biases which cannot be corrected by using multiple workers. We investigate a game-theoretic bonus scheme, called Peer Truth Serum (PTS), to overc...
متن کاملBeyond the Bayesian Truth Serum: The Knowledge Free Peer Prediction Mechanism
The elicitation of private information from individuals is crucially important to many tasks, ranging from scientific research to corporate decision-making. Eliciting private information is particularly challenging when objective truth is inaccessible when there is no “anwer key” available. To address this challenge, we present the Knowledge Free Peer Prediction mechanism (KFPP). KFPP induces t...
متن کامل